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The invention relates to a method of segmenting a first image feature in a first 
video image from an adjacent second image feature in the first video image. 

The invention further relates to a segmentation system for segmenting a first 
image feature in a first video image from an adjacent second image feature. 
5 The invention further relates to a motion estimator at pixel resolution for 

estimating a motion vector field, comprising such a segmentation system. 

The inveation further relates to an image processing apparatus comprising: 
■ receiving means for receiving a signal representing a series of video images; 

- such a motion estimator at pixel resolution for estimating a motion vector field from the 
10 video images; and 

- amotion compensated image processing unit for determining processed images on basis of 
the video images and the motion vector field. 



* 5 An embodiment of the method of the kind described in the opening paragraph 

is known from US patent 6,075,875, This method includes obtaining a motion representation, 
e.g. motion vectors, of corresponding pixels in a selected video image and a preceding video 
image to form motion-segmented video image features. Video image features are also 
segmented according to their spatial image characteristics to form spatially-segmented video 

20 image features. Finally the video image features are jointly segmented as a weighted 
combmation of the motion-segmented video image features and the spatially-segmented 
video image features. A disadvantage of the method according to the prior art is the relatively 
complex method of combining motion-segmented video image features and the spatially- 
segmented video image features. Notice that two different quantities, i.e. a motion based 

25 quantity and a Spatial image characteristic, are weighted. Weighting different quantities is 
unusual. 
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It is an object of the invention to provide a method of the kind described in the 
opening paragraph which is pixel accurate and relatively easy. 

This object of jhe invention is achieved in that the method of segmenting a 
first image feature in a first video image from an adjacent second image feature in the first 
5 video image* the first image feature having plural pixels with respective values of an image 
property substantially being in a first range of values and having motion relative to the 
second image feature between the first video image and a second video image, and the 
second image feature having plural pixels with respective values of the image property 
substantially being in a second range of values being different from the first range of values, 
10 comprises; 

- dividing the first video image into blocks of pixels; 

- estimating motion vectors for the respective blocks of pixels; 

- segmenting the first video image into a first group of connected blocks of 
pixels and a second group of connected blocks of pixels by classifying the blocks of pixels on 

15 basis of the motion vectors of the respective blocks of pixels; and succeeded by 

- segmenting the first image feature from the second image feature by means 
of a pixel-based segmentation of a portion of the blocks of pixels of the first and second 
group of connected blocks of pixels, which have been determined to be positioned at a border 
between the first and second group of connected blocks of pixels, on basis of the respective 

20 values of the image property. 

An advantage of the method according to the invention is that the two 
segmentation operations are applied on the appropriate locations. The result of the block- 
based motion segmentation is used as input for the pixel-based segmentation on basis of the 
image property. The pixel-based segmentation is applied to recover object boundaries, i.e. 

25 image feature boundaries on a small subset of the video image, i.e. in the blocks at the border 
between the first and second group of connected blocks of pixels. This is an efficient strategy 
resulting in efficient computing resource usage, since the detailed pixel-based segmentation 
process is only applied to the blocks at the boundaries and not to all blocks. The block-based 
motion segmentation and the pixel-based segmentation are applied sequentially: No complex 

30 weighting factors are required to tune between the results of the segmentation approaches as 
in the prior art* 

In an embodiment of the method according to the invention, segmenting the 
first video image into the first group of connected blocks of pixels and the second group of 
connected blocks of pixels is based on a motion model. The advantage of this embodiment is 



ox . u UU . CXXJC j.cf'. icf rniUJ.ro UXi i«- »*ja Lrtgsos _ _ 

i »r\ • *WLo r . O^ — 

PHNL020744EPP 009 31. 07. 2002 14:18 

° o 

3 31.07.2002 
that It is a robust method. Relatively much data, i.e. most or all motion vectors of test groups 
of connected blocks for the first group of connected blocks of pixels are applied to estimate 
the appropriate configuration of blocks which belong to fixe first image feature. In other 
words it is a type of region fitting. Many segmentation methods only take blocks at the 
5 border of a segment into account to decide whether the blocks correspond to the particular 
segment or not. The motion model might be based on rotation and/or translation. However 
preferably the segmenting is based on an affine motion model, 

In an embodiment of the method according to the invention, segmenting the 
first video image into the first group of connected blocks of pixels and 1he second group of 
10 connected blooks of pixels comprises: 

- creating a first initial group of connected blocks of pixels for the first group 
of connected blocks of pixels, the first initial group of connected blocks of pixels comprising 
a particular block of pixels; 

- determining a first motion model for the first initial group of connected block 

15 ofpixels; 

- calculating a first match error between the motion vector corresponding to 
the particular block ofpixels being estimated during estimating motion vectors for the 
respective blocks of pixels of the first video image and the motion vector corresponding to ; 
me particular block of pixels on basis of the first motion model; 

20 - calculating a second motion model for a test group of connected blocks of 

pixels, based on the first initial group of connected blocks ofpixels, but excluding the 
particular block ofpixels; 

- calculating a second match error between the motion vector corresponding to 
the particular block ofpixels being estimated during estimating motion vectors for me 

25 respective blocks ofpixels of the first video image and the motion vector corresponding to 
the particular block of pixels on basis of the second motion model; 

- deciding whether the particular block ofpixels corresponds to the first group 
of connected blocks of pixels or not on basis of 1he first and second match error. 

An advantage of this embodiment according to the invention is that it allows 
30 for a scanning approach. That means that block by block, e.g. starting at the left top towards 
the right bottom each block is tested iterative. Testing means that for each block it is 
evaluated to with which other blocks it should be merged to eventually foimthe group of 
blocks corresponding to one of the image features, The advantage of the scanning approach is 
that the method can be implemented relatively easy. 



«. JU u.—c ffllurewr,fc 010 31.07.2002 14:1 

PHNL020744EPP 

4 W 31.07.2002 

In an embodiment of the method according to the invention the pixel-based 
segmentation is based on a spatial color model, hi other words the image property on which 
the pixel-based segmentation is based, is color. Color is a relatively good cue for 
segmentation, m this embodiment the two segmentation approaches are applied on the 
5 appropriate scale resolution: color is applied for high frequencies and motion segmentation 
for low frequencies. Since color differences are caused by texture and motion differences 
caused by different velocities of objects in the scene being captured. 

hi an embodiment of the method according to the invention the pixeM>ased 
segmentation is based on a spatial luminance model, hi other words the image property on 
10 which the pixel-based segmentation is based, is luminance. Luminance is a relatively good 

cue for segmentation. 

hi an embodiment of the method according to the invention a step in values of 
the image property is detected in a first blook of the portion of the blocks of pixels of the first 
group of connected blocks of pixels, which have been determined to be positioned at the 
1 5 border between the first and second group of connected blocks of pixels. A step in values 

corresponds to the edge of the first image feature. Preferably the step is detected by means of: 
_ calculating for the pixels of the first block a first mean value of the image 

property; 

- calculating a first difference measure on basis of the first mean value and the 
20 respective values of the pixels of the first block; 

. calculating for the pixels of a second block of pixels a second mean value of 
the image property, the second block of pixels corresponding to the second group of pixels 
and being connected to file first block; 

- calculating a second difference measure on basis of the second mean value 
25 and the respective values of the pixels of the second block; 

- creating a first test group of pixels on basis of the first block but excluding a 
particular pixel and creating a second test group of pixels on basis of the second block and 
comprising the particular pixel; 

- calculating for the pixels of the first test group a third mean value of the 

30 image property; 

_ calculating athird difference measure on basis of the third mean value and 

the respective values of the pixels of the first test group; 

- calculating for the pixels of the second test group a fourth mean value of the 

image property; 
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- calculating a fourth difference measure on basis of the fourth mean value and 
the respective values of the pixels of the second test group; 

- deciding whether the particular pixel belongs to the first image feature or the 
second image feature on basis of the first, second, third and fourth difference measure. 

5 The advantage of this embodiment is that it is a robust method. Relatively much data, i.e. 
most or all pixels of me first and second block of pixels are applied to estimatethe 
appropriate configuration of pixels which belong to the first image feature. This is a so-called 
region fitting approach. 

It is another object of foe invention to provide a segmentation system of the 
10 kind described in the opening paragraph which is arranged to provide pixel accurate 
segmentations and which is relatively easy. 

This object of the invention is achieved in that me segmentation system for 
segmenting a first image feature in a first video image from an adjacent second image feature 
in Uie first video image, the first image feature having plural pixels with respective values of 
15 an image property substantially being In a first range of values and having motion relative to 
the second image feature between the first video image and a second video image, and the 
second image feature having plural pixels with respective values of the image property 
substantially being in a second range of values being different from 1he first range of values, 
comprises; 

20 " dividing means for dividing the first video image into blocks of pixels; 

- ablock-based motion, estimator for estimating motion vectors for me, 
respective blocks of pixels; 

- a motion segmentation unit for segmenting me first video image into a first 
group of connected blocks of pixels and a second group of connected blocks of pixels by 

25 classifying foe blocks of pixels on basis of foe motion vectors of the respective blocks of 
pixels; and 

" a Pixel-based segmentation unit for segmenting foe first image feature from 
the second image feature by means of a pixel-based segmentation of a portion of the blocks 
of pixels of the first and second group of connected blocks of pixels, which have been 
30 detemuued to be positioned at a border between foe first and second group of connected 
blocks of pixels, on basis of the respective values of foe image property. 

It is another object of the invention to provide a motion estimator at pixel 
resolution for estimating a pixel accurate motion vector field, which is relatively easy. 
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This object of the invention is achieved in that the pixel accurate motion 
estimator comprises the segmentation system as claimed in claim 9. Alternatively, the pixel 
accurate motion estimator is provided with results being calculated by the segmentation 
system. 

5 It is advantageous to apply an embodiment of the pixel accurate motion 

estimator according to the invention in an image processing apparatus as described in the 
opening paragraph. The image processing apparatus may comprise additional components, 
e.g. a display device for displaying the processed images or storage means for storage of the 
processed images. The motion compensated image processing unit might support one or 
10 more of the following types of image processing: 

- De4nterlacing: Interlacing is the common video broadcast procedure for 
transmitting the odd or even numbered image lines alternately, De-interlaoing attempts to 
restore the full vertical resolution, ie. make odd and even lines available simultaneously for 



15 - Up-conversion: From a series of original input images a larger series of 

output images te calculated. Output images are temporally located between two original input 



- Temporal noise reduction- This can also involve spatial processing, resulting 
in spatial-temporal noise reduction; and 
20 - Video compression, i.e. encoding or decoding, e.g. according to the MPEG 

standard or H26L standard. 

Modifications of the method and variations thereof may correspond to 
modifications and variations thereof of the segmentation system, the pixel accurate motion 
estimator and of the image processing apparatus described. 

25 

These and other aspects of the method, the segmentation system, the pixel 
accurate motion estimator and of the image processing apparatus according to the invention 
will become apparent from and will be elucidated with respect to the implementations and 
30 embodiments described hereinafter and with reference to the accompanying <Jrawings 9 
wherein; 

Fig, 1 schematically shows an embodiment of the segmentation system 
according to the invention; 
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Fig. 2A schematically shows an initial configuration of blocks for the block- 
based motion segmentation: 

Fig. 2B schematically shows an updated configuration of blocks of Fig. 2A 
after one modification; 

5 Kg. 2C schematically shows a result configuration of blocks of the block- 

based motion segmentation; 

Kg. 3A schematically shows input blocks for the pixel-based segmentation; 
Kg. 3B schematically shows an initial edge for a horizontal block pair for the 
pixel-based segmentation; 

10 3C schematicaU y shows *e detected edge for the horizontal block pair of 

Fig3B; 

Fig. 3D schematically shows an initial edge for a vertical block pair for the 
pixel-based segmentation; 

Kg. 3E schematically shows me finally detected edge; 
15 Fi S- 4A schematically shows the inputs and output of a motion estimator for 

estimating a pixel accurate motion vector field; 

Fig. 4B schematically shows the inputs and output of an alternative motion 
estimator for estimating a pixel accurate motion vector field; and 

Fig. 5 schematically shows elements of an image processing apparatus 
20 according to the invention. 

Same reference numerals are used to denote, similar parts throughout the figures. 



25 



Kg. 1 schematically shows an embodiment of the segmentation system 100 
according to the invention. The segmentation system 100 is arranged to segment a first image 
feature 214 in a first video image from an adjacent second image feature 216 in the first 
video image. The first image feature 214 has plural pixels with respective color values 
substantially being in a first range of values and the second image feature 216 has plural 
pixels with respective color values substantially being in a second range of values being 
30 different from the first range of values. The first image feature 214 has motion relative to the 
second image feature 216 between the first video image and a second video image. The 
second video image might be succeeding or preceding the first video image. The 
segmentation system 100 receives a signal representing video images at the input connector 
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108 and provides a pixel acourate segmentation result S p st the output connector 110, The 
segmentation system 200 comprises; 

_ a block-based motion estimator 102 for estimating motion vectors. This 
motion estimator 102 is airanged to estimate motion vectors 218-230 for the respective 
5 blocks in which the first video image has been divided by the splitting unit 103. This dividing 
means that the pixels of the first video image are clustered into bloclcs of pixels. Typically a 
block of pixels comprises 8*8 pixels. Preferably the motion estimator 102 is as described in 
the article "True-Motion Estimation with 3-D Recursive Search Block Matching" by G. de 
Haan et al. in IEEE Transactions on circuits and systems for video technology, vol.3, no.S, 
10 October 1993, pages 368-379. The motion estimator 102 provides ablock-based motion 

vector field M B to; 

, a motion segmentation unit 104 which is arranged to segment the first video 
image into a first group of connected blocks of pixels 204C (see Fig. 2C) and a second group 
of connected blocks of pixels 206C (see Fig. 2C) by classifying ine blocks of pixels on basis 
15 of the motion vectors 218-230 of the respective blocks ofpixels. The segmentation unit 104 

provides a block-based segmentation result S* to; 

- a pixel-based segmentation unit 106 which is arranged to segment 1he first 
image feature 214 from the second image feature 216 by means of a pixel-based 
segmentation of aportion of the blocks 302-306 ofpixels of the first and second group of 
20 connected blocks ofpixels 204, 206, which have been determined to be positioned at a border 
between the first and second group of connected blocks of pixels 204, 206. The pixel-based 
segmentation unit 106 is designed to perform the segmentation on basis of color values of 
pixels. 

The behavior of the motion segmentation unit 104 will be described in more 
25 detail in connection with Figs. 2A-2C and me behavior of the pixel-based segmentation unit 
106 will be described in more detail in connection with Figs. 3A-3E. 

The block-based motion estimator 102, the motion segmentation unit 104 and 
the pixel-based segmentation unit 106 of the segmentation system 100 may be implemented 
using one processor. Normally, these functions are performed under control of a software 
30 program product During execution, normally the software program product is loaded into a 
memory, like a RAM, and executed from there. The program may be loaded from a 
background memory, like a ROM, hard disk, or magnetically and/or optical storage, or may 
be loaded via a network like Internet Alternatively an application specific integrated circuit 
provides the disclosed functionality. 
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la Kg. 2A a motion vector field 200 is depicted. The motion vector field 200 
comTmsesanmberofmotio^^ ThesB 
Mocks of pixels have been clustered intd 6 initial groups of connected blocks of pixels 202A- 
212A. Or in other words, Fig. 2A schematically shows an initial configuration of blocks for 
the block-based motion segmentation. The number of initial groups of connected blocks 
corresponds to the maximum number of image features, i.e. objects which one want to 
segment in the video image. Xh this exemplary cases each initial group of connected blocks of 
pixels 202A-212A comprises 5*5 blocks. All blocks of pixels of a particular initial group of 
connected blocks have the same classification label. By means of a scanning approach all 
blocks will be evaluated. That means mat appropriate clusters or groups of connected blocks 
of pixels 202C-206C (See Kg. 2C) are determined. Such a group of connected blocks of 
pixels 202C-206C corresponds to one of the image features 214, 216 in the first video image. 
Finding me appropriate groups is achieved in a number of iterations wherein tor me blocks is 
tested to which group of connected blocks of pixels 202-206C it belongs to. That might be in 
a fixed number of iterations. Alternatively the process of evaluating the various intermediate 
groups of conneoted blocks 202B-212B is stopped if modifications of intermediate groups of 
connected blocks 202B-212B do not have any positive effect. The process of evaluating is 
bounded by a topological constraint. Only blocks at the border of a connected groups of 
blocks 202A-212A, 202B-212B are tested. Notice mat a block might become located at the 
border of an intermediate group of blocks 202B-212B after a number of iterations. Only 
applying blocks at the border is different from the well-known k-means approach which does 
not have mis constraint. The evaluation of a particular block of pixels 232 comprises a 
number of steps. This is described for the particular block of pixels 232 during the first scan 
(See Fig. 2A): 

- creating a first initial group of connected blocks of pixels 204A comprising 
toe particular block of pixels 232. 

" determining a first motion model for the first initial group of connected block 
of pixels 204A; 

- calculating a first match error between the motion vector corresponding to 
the particular block of pixels 232 being estimated during estimating motion vectors for the 
respective blocks of pixels of the first video image and toe motion vector corresponding to 
the particular block of pixels 232 on basis of the first motion model. Preferably the match 
error is based on the Euclidean distance between the motion vectors; 
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- calculating a second motion model for a test group of connected blocks of 
pixels 204B, based on the first initial group of connected blocks of pixels 2Q4A, but 
excluding the particular block of pixels 232. (See 204B in Fig. 2B)j 

- calculating a second match error between the motion vector corresponding to 
5 the particular block of pixels 232 being estimated during estimating motion vectors for the 

respective blocks of pixels of the first video image and the motion vector corresponding to 
the particular block of pixels 232 on basis of the second motion model; 

_ deciding whether the particular block of pixels 232 corresponds to the first 
group of connected blocks of pixels 204C or not on basis of the first and second match error. 
10 This means that it is tested -whether the particular blook of pixels 232 matches better with the 
first initial group of connected blocks of pixels 204A or with the neighboring group of 
connected block of pixels 202A. hi this case it appeared that the particular block of pixels 
232 better matches with the latter group of blocks 202A and hence the neighboring group of 
connected blocks of pixels 202A is extended with the particular block 232 resulting hi the 
15 intermediate group of blocks 202B. The first initial group of blocks 204A is adapted into an 
intermediate group of blocks 204B excluding the particular block 232 (See Fig. 2B). 

Alternatively, additional match errors are calculated and taken into account to 
decide to which of the initial groups of blocks 202A-212A the particular block of pixels 232 
belongs to. E.g. athird match error is calculated for a second initial group of connected 
20 blocks of pixels 202A as depicted in Fig. 2A and a fourth match error for a second test group 

of connected blocks of pixels 202B as depicted in Fig. 2B. 

The test whether the particular block of pixels 232 matches better with the first 
initial group of connected blocks of pixels 204A or with the neighboring group of connected 
block of pixels 202A can also be performed by means of evaluating with the first initial 
25 group of connected blocks of pixels 204A and with foe neighboring group of conneoted block 
of pixels 202A. That means without adapting these groups by moving the particular group of 
pixels 232 from the first initial group of connected blocks of pixels 204A to the neighboring 
group of connected block of pixels 202A. In that case motion models are calculated for both 
groups of conneoted blocks. 
30 Fig. 2B schematically shows an updated configuration of blocks for the block- 

based motion segmentation after one modification. This modification can be executed 
directly or after all relevant blocks of the motion vector field have been evaluated. With 
relevant is meant that the blocks are positioned at the border of a group of connected blocks 
of pixels. 
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Ifcis shovmu^t aparticular block of pixels 232 is moved from a first initial 
group of connected blocks of pixels 2Q4A to a horizontally positioned neighboring group of 
blocks 202A. However a movement in vertical direction is also possible, e.g. a movement of 
another block of pixels from the first initial group of blocks 204A to the group of blocks of 
5 pixels 210A beneath, 

After anumber of iterations a number of final groups of blocks of pixels 
202C-206C is found. These final groups of blocks of pixels 2Q2C- 206C are the output of the 
block-based motion segmentation. Fig. 2C schematically shows this output, i.e. the result 
configuration of the block-based motion segmentation. This output is provided to the pixel- 
10 based segmentation unit 106 for further processing: detecting the actual borders of the first 
image feature 214 and the second image feature 216. 

Preferably the block-based motion segmentation result for one image pair, e.g. 
the first image and the second image is applied for initialization of ihe block-based motion 
segmentation of a succeeding image pair. 
15 Fig, 3A schematically shows input blocks 302-306 for the pixel-based 

segmentation. The motion segmentation result indicates that one of the blocks belongs to a 
first image video feature 214 and is labeled as such 1 and that two of the blocks 304 and 306 
belong to a second image video feature 216 and are labeled as such 2. Pairs of neighboring 
blocks are evaluated to find the actual edge of the first image feature 214. 

20 Kg. 3B schematically shows an initial edge 308 for a horizontal pair of 

neighboring blocks 302 and 304. In a similar way as described above for hie block-based 
motion segmentation the pixel-based segmentation is performed: with similar is meant that 
for pixels is evaluated to which group of connected pixels they correspond. This is done on 
basis of a spatial color model which is based on the following assumptions: Comprised by the 

25 pair of neighboring blocks 302 and 304 there is a first group of connected pixels with color 
values substantially being in a first range of values and a second group of pixels with color 
values substantially being in a second range of values. Besides that there is a transition or 
step in color values between the first group of connected pixels and the second group of 
connected pixels. That means that for pixels of the pair of neighboring blocks 302 and 304 is 

30 evaluated to which of the two groups of connected pixels the respective pixels correspond. 
The evaluation is done by means of an iterative process. That might be in a fixed number of 
iterations. Alternatively the process of evaluating the various intermediate groups of 
connected pixels is stopped if modifications of intermediate groups of connected pixels do 
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not have any positive effect The evaluation for a particular pixel comprises the foUomng 
steps: 

- calculating for the pixels of the first block of pixels 302 of the pair of 
neighboring blocks 302, 304 a first mean color value; 

5 - calculating a first difference measure on basis of the first mean color value 

and the respective values of the pixels of the first block 302; 

~ calculating for the pixels of the second block of pixels 304 of the pair of 
neighboring blocks 302, 304 a second mean color value; 

- calculating a second difference measure on basis of the second mean color 
10 value and the respective values of the pixels of the second block of pixels 3 04; 

« creating a first test group of pixels on basis of the first block 302 but 
excluding the particular pixel and creating a second test group of pixels on basis of the 
second block 304 and comprising the particular pixel; 

- calculating for the pixels of the first test group a third mean color value; 
IS - calculating a third difference measure on basis of the third mean color value 

and the respective values of the pixels of the first test group; 

- calculating for the pixels of the second test group a fourth mean color value; 

- calculating a fourth difference measure on basis of the fourth mean color 
value and the respective values of the pixels of the second test group; 

20 - deciding whether the particular pixel belongs to the first group of connected 

pixels or the second group of connected pixels, on basis of the first, second* third and fourth 
difference measure. 

After a number of iterations the eventual first group of pixels comprises pixels 
which correspond to the first image feature and the eventual second group of pixels 

25 comprises pixels which correspond to the second image feature. Fig- 3C schematically shows 
the detected edge 308 for the horizontal block pair of Fig 3B. 

After the evaluation of the horizontally positioned pair of neighboring blocks 
302, 304 has been completed a similar evaluation is started for a vertically positioned pair of 
neighboring blocks 302, 306- Fig. 3D schematically shows an initial edge 310 for this 

30 vertically positioned pair of neighboring blocks 302, 306, Fig. 3E schematically shows the 
finally detected edge 3 10- Notice that the evaluation of the horizontally positioned pair of 
neighboring blocks 302, 304 and the vertically positioned pair of neighboring blocks 302, 
306 might be performed simultaneously. 
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Although only color is discussed, it should be noted that other image 
properties can be applied for the pixel-based segmentation, e.g. luminance, or a combination 
of color and luminance or derived properties such as differences in color/Iuminanoe values 
between neighboring pixels. 
5 Pig. 4A schematically shows the input and outputs of a motion estimator 400 

for estimating a pixel accurate motion vector field. A motion vector field M 3 as being 
calculated by die block-based motion estimator 102 as described in connection -with Fig. 1 is 
provided at input connector 404. Optionally the motion models as being calculated by the 
motion segmentation unit 104 are also provided. Because of the block resolution of the 
10 motion vector field Af * the motion vectors 21 8-230 are not correct for all pixels of the first 
video image. However at another input connector 406 of the motion estimator 400 also a 
pixel accurate segmentation result S p is provided by the segmentation system 100 as 
described in connection with Fig. 1 . By means of combining the information which is 
provided a pixel accurate motion vector field can be determined. Especially pixels of blocks 

15 being located at the borders of the image features 214, 216 might be erroneous. Preferably 
motion vectors are assigned to these respective pixels which correspond to motion vectors 
being estimated tor neighboring blocks. The selection of the appropriate neighboring block 
and thus motion vector is determined by the segmentation result •S'* . In general, to a portion 
of the pixels of a block being located at the border of an image feature 214, 216 a motion 

20 vector is assigned which is equal to a motion vector of a block corresponding to a first image 
feature 214 and another to another portion of pixels another motion vector is assigned which ■ - 
is equal to a motion vector of a block corresponding to a second image feature 216 being 
adjacent to the first image feature 214. 

Fig. 4B schematically shows the inputs and output of an alternative motion 

25 estimator 401 for estimating a pixel accurate motion vector field. Video images are provided 
at the input connector 402 and at another input connector 406 of the motion estimator 401 
also a pixel accurate segmentation result S p is provided by the segmentation system 100 as 
described in connection with Fig. 1. The pixel accurate motion estimator 401 is arranged to - 
calculate motion vectors tor the groups of pixels corresponding to the respective image 

30 features 214, 216 as segmented by the segmentation system 100 by means of comparing the 
pixels of these groups of pixels with coiiesponding pixels of preceding or succeeding video 
images. Preferably the comparing is based on match errors conresponding to the sum of 
absolute pixel value differences. 
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Fig. 5 schematically shows elements of an image processing apparatus 500 
according to the invention. The processing apparatus 500 comprises: 

- a receiving unit 502 for receiving a signal representing video images to "be 
displayed after some processing has been performed, The signal may be a broadcast signal 

5 received via an antenna or cable but may also be a signal from a storage device like a VCR 
(Video Cassette Recorder) or Digital Versatile Disk (DVD). The signal is provided at the 
input connector 510. 

- a processing unit 504 comprising a segmentation system 100 and a motion 
estimator 401 as described in connection with Pig, 1 and Fig. 4A> respectively; 

10 -a motion compensated image processing unit 506; and 

« a display device 508 for displaying the processed images. This display device 
508 is optional. 

The motion compensated image processing unit 506 requites images and 
motion vectors as its input. The motion compensated image processing unit 506 might 

15 support one or more of the following types of image processing: de-interlacing; up- 
conversion; temporal noise reduction; and video compression, 

It should be noted that the above-mentioned embodiments illustrate rather than 
limit the invention and that those skilled in the art will be able to design alternative 
embodiments without departing from the scope of the appended claims. In the claims* any 

20 reference signs placed between parentheses shall not be constructed as limiting the claim. 
The word 'comprising 7 does not exclude the presence of elements or steps not listed in a 
claim. The word "a" or "an" preceding an element does not exclude the presence of a 
plurality of such elements. The invention can be implemented by means of hardware 
comprising several distinct elements and by means of a suitable programmed computer* In 

25 the unit olaims enumerating several means, several of these means can be embodied by one 
and the same item of hardware. 



PHNL020744EPP 



CLAIMS: 



o 



021 31.07.2002 14:23 



15 



o 



31.075002 



1- A method of segmenting a first image feature (214) in a first video image from 

an adjacent second image feature (216) in the first video image, die first image feature (2X4) 
having plural pixels with respective values of an image property substantially being in a first 
range of values and having motion relative to the second image feature (216) between the 
5 first video image and a second video image, and the second image feature (216) having plural 
pixels with respective values of the image property substantially being in a second range of 
values being different from the first range of values, the method comprising; 

- dividing the first video image into blocks of pixels; 

- estimating motion vectors (218-230) for the respective blocks of pixels; 
10 - segmenting the first video image into a first group of oonnected blocks of 

pixels (204C) and a second group of connected blocks of pixels (2Q6C) by classifying the 
blocks of pixels on basis of the motion vectors (218-230) of the respective blocks of pixels; 
and succeeded by segmenting the first image feature (214C) from the second image feature 
(216Q by means of a pixel-baaed segmentation of a portion of the blocks of pixels of the 
15 first and second group of connected blocks of pixels (204C, 206C), which have been 
determined to be positioned at a border between die first and second group of connected 
blocks of pixels (204C, 206C), on basis of the respective values of the image property. 

2 - Amethod as claimed in claim i, whereby segmenting the first video image 
20 into the first group of connected blocks of pixels (204C) and the second group of connected 

blocks of pixels (206Q is based on a motion model. 

3 - A method as claimed in claim 2, whereby segmenting the first video image 
into the first group of connected blocks of pixels (204C) and the second group of connected 

25 blocks of pixels (206C) is based on an affine motion model. 



4 - A method as claimed in claim 2 or 3, whereby segmenting the first video 

image into the first group of connected blocks of pixels (204C) and the second group of 
connected blocks Of pixels (206C) comprises: 
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- creating a first initial group of connected blocks of pixels (204A) for the first 
group of connected Mocks of pixels (204Q* the first initial group of connected blocks of 
pixels (204A) comprising aparticular block of pixels (232); 

- determining a first motion model for the first initial group of connected block 
5 of pixels (204A); 

- calculating a first match error between the motion vector corresponding to 
the particular block of pixels (232) being estimated during estimating motion vectors (218- 
230) for the respective blocks of pixels of the first video image and the motion vector 
corresponding to the particular block of pixels (232) on basis of the first motion model; 

10 - calculating a second motion model for a test group of connected blocks of 

pixels (204B), based on the first initial group of connected blocks of pixels (204A), but 
excluding the particular block of pixels (232); 

- calculating a second match error between the motion vector corresponding to 
the particular block of pixels (232) being estimated during estimating motion vectors (218- 

1 5 230) for the respective blocks of pixels of the first video image and the motion vector 

corresponding to the particular block of pixels (232) on basis of the second motion model; 

- deciding whether the particular block of pixels (232) corresponds to the first 
group of connected blocks of pixels (204C) or not on basis of the first and second match 
error, 

20 

5 w A method as claimed in claim 1, whereby the pixel-based segmentation is 

based on a spatial color model. 

6. A method as claimed in claim 1, whereby the pixel-based segmentation is 
25 based on a spatial luminance model. 

7, a method as claimed in claim S or 6 9 whereby a step in values of the image 
property is detected in a first block (302) of the portion of the blocks of pixels of the first 
group of connected blocks of pixels (204C), which have been determined to be positioned at 

30 the border between the first and second group of connected blocks of pixels (204C, 206C). 



g. A method as claimed in claim 7 ? whereby the step is detected by means of: 

- calculating for the pixels of the first block (302) a first mean value of the 
image property; 
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- calculating a first difference measure on basis of the first mean value and the 
respective values of the pixels of the first block (302): 

- calculating for the pixels of a second block of pixels (304) a second mean 
value of the image property, the second block (304) of pixels corresponding to the second 

5 group of pixels and being connected to the first block (302); 

- calculating a second difference measure on basis of the second mean value 
and the respective values of the pixels of the second block (304); 

- creating a first test group of pixels on basis of the first block (302) but 
excluding a particular pixel and creating a second test group of pixels on basis of the second 

10 block (304) and comprising the particular pixel; 

- calculating for the pixels of the first test group a third mean value of the 
image property; 

- calculating a third difference measure on basis of the third mean value and 
the respective values of the pixels of the first test group; 

15 - calculating for the pixels of the second test group a fourth mean value of the 

image property; 

- calculating a fourth difference measure on basis of the fourth mean value and 



- deciding whether the particular pixel belongs to the first image feature (214) 
20 or the second image feature (216) on basis of fixe first, second, third and fourth difference 
measure. 



9. A segmentation system (100) for segmenting a first image feature (214) in a 

first video image from an adjacent second image feature (216) in the first video image, the 

25 first image feature (214) having plural pixels with respective values of an image property 
substantially being in a first range of values and having motion relative to the second image 
feature (216) between the first video image and a second video image, and the second image 
feature (216) having plural pixels with respective values of the image property substantially 
being in a second range of values being different from the first range of values, the 

30 segmentation system (1 00) comprising; 

- dividing means (103) for dividing the first video image into blocks of pixels; 

- a block-based motion estimator (102) for estimating motion vectors (218- 
230) for the respective blocks of pixels; 
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- a motion segmentation unit (104) for segmenting the first video image into a 
first group of connected blocks of pixels (204C) and a second group of connected blocks of 
pixels (206) by classifying the blocks of pixels on basis of the motion vectors (218-230) of 
the respective blocks of pixels; and 
5 - a pixel-based segmentation unit (106) for segmenting the first image feature 

(214) from the second image feature (216) by means of a pixel-based segmentation of a 
portion of the blocks of pixels of the first and second group of connected blocks of pixels 
(204C, 206C), which have been determined to be positioned at a border between, the first and 
second group of coanected blocks of pixels (204C, 206C), on basis of the respective values 
10 of the image property. 

10. A motion estimator at pixel resolution (400, 401) for estimating a motion 

vector field, comprising the segmentation system (100) as claimed in claim 9. 

15 1 L A motion estimator at pixel resolution (400) as claimed in claim 10 being 

arranged to assign the motion vectors (218-230) estimated for the respective blocks to the 
respective pixels of the first video image on basis of the segmenting the first image feature 
(214) from the second image feature (216). 

20 12. A motion estimator at pixel resolution (401)as claimed in claim 10 being 

arranged to estimate a new motion vector for the first image feature (214) by means of 
comparing pixels of the first image feature (214) with corresponding pixels of the second 
Video image. 



25 13, An image processing apparatus (500) comprising: 

- receiving means (502) for receiving a signal representing a series of video 

images; 

- a motion estimator at pixel resolution (504) as claimed in claim 10 for 
estimating a motion vector field from the video images; and 

30 - a motion compensated image processing unit (506) for determining 

processed images on basis of the video images and the motion vector field. 



14. An image processing apparatus (500) as claimed in claim 13, whereby the 

motion compensated image processing unit (506) is designed to perfbtm video compression. 
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15. An image processing apparatus (500) as claimed in claim 13, whereby &© 
motion compensated image processing unit (506) is designed to reduce noise in the series of 
images. 

5 

16. An image processing apparatus (500) as claimed in claim 13, whereby the 
motion compensated image processing unit (506) is designed to de-interlace the series of 
images. 



10 17. An image processing apparatus (500) as claimed in claim 13, characterized ii 

that the motion compensated image processing unit (506) is designed to perform an up- 
conversion. 
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ABSTRACT; 



A segmentation system (100) for segmenting a first image feature (214) in a 
first video image from an adjacent second image feature (216) in the first video image on 
basis of motion and on an image property like color or luminance. The segmentation system 
(100) comprises; a block-based motion estimator (102) for estimating motion vectors (218- 

5 230) for blocks of pixels; a motion segmentation unit (104) fox segmenting the first video 
image into a first group of connected blocks of pixels (204) and a second group of connected 
blocks of pixels (206) on basis of the motion vectors (218-230) of the respective blocks of 
pixels; and a pixel-based segmentation unit (106) for segmenting the first image feature (214) 
ftom the second image feature (214) by means of a pixel-based segmentation of a portion of 

10 the blocks of pixels of the first and second group of connected blocks of pixels (214 and 
21 6)* on basis of the respective values of the image property. 
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